The term “Big Data” is used as a blanket term for any collection of datasets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to spot business trends, determine quality of research, prevent diseases, link legal citations, combat crime, and determine real-time roadway traffic conditions.
Big data is difficult to work with using most relational database management systems and desktop statistics and visualization packages, requiring instead massively parallel software running on tens, hundreds, or even thousands of servers. What is considered “Big data” varies depending on the capabilities of the organization managing the set, and on the capabilities of the applications that are traditionally used to process and analyze the data set in its domain. Big Data usually includes data sets with sizes beyond the ability of commonly used software tools to manage, and process the data within a tolerable elapsed time.
As used herein, a large database is a columnar or hybrid columnar database which stores data tables as sections of columns of data rather than as rows of data and that includes at least one petabyte of data. Such large databases are currently used for a variety of applications in which very large amounts of data is stored, queried, aggregated, analyzed and searched such as business data analytics applications. Modern business data analytics applications compute aggregates over a large amount of data to roll-up information along an increasing number of dimensions such as geographic regions, demography, users, products, etc. Traditionally, online business analytics databases executed such queries by performing sequential scans over a significant portion of the database. As a result of the increasing sizes and dimensions, and increasing importance of interactive query response times of today's large analytics databases, querying the database by scanning the entire large database is not feasible. In addition to the size of large databases other factors make low latency querying of large databases with known techniques difficult. For example, in online analytics databases the percentage of database queries that are ad hoc in nature is high, which makes the creation of an index for the large database difficult. The large number of dimensions render techniques such as pre-computed cubes very space and computationally exorbitant.
Currently, these large analytics databases are stored on traditional data storage devices such as hard disk drives and the like. Recently, in an attempt to improve the performance of the large columnar databases, some large databases have been stored on high performance storage devices such as solid state devices or flash memory and the like. While storing large databases on high performance storage devices increases the speed of some queries on the large databases, the increased performance comes at a high cost as high performance storage devices are much more expensive than traditional data storage devices.